Papers by Aryo Pradipta Gema
Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering (2025.naacl-long)
Copied to clipboard
Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, Pasquale Minervini
| Challenge: | Large language models store factual knowledge in their parameters but their parametric knowledge can conflict with the information provided in the context. |
| Approach: | They propose a training-free representation engineering method that uses pre-trained sparse auto-encoders to control the knowledge selection behaviour of large language models. |
| Outcome: | The proposed method can control the use of both knowledge sources to resolve knowledge conflict in open-domain question-answering tasks surpassing existing representation engineering methods (+10%) and contrastive decoding methods (+5%). |
CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Mathematical reasoning remains a significant challenge for large language models (LLMs), despite advances in prompting techniques such as Chain-of-Thought (CoT). |
| Approach: | They propose a framework that enhances reasoning through two stages: Symbolic Conversion and Reasoning Execution. |
| Outcome: | The proposed framework outperforms traditional CoT on six out of seven benchmarks across four LLMs. |
PiCSAR: Probabilistic Confidence Selection and Ranking for Reasoning Chains (2026.findings-acl)
Copied to clipboard
Joshua Ong Jun Leang, Zheng Zhao, Aryo Pradipta Gema, Sohee Yang, Wai-Chung Kwan, Xuanli He, Wenda Li, Pasquale Minervini, Eleonora Giunchiglia, Shay B Cohen
| Challenge: | Recent studies show that large reasoning models (LLMs) achieve strong performance on complex reasoning tasks. |
| Approach: | They propose a method that scores each candidate generation using the joint log-likelihood of the reasoning and final answer. |
| Outcome: | The proposed method outperforms baselines with 2x fewer samples in 20 out of 25 comparisons. |
Self-Training Large Language Models for Tool-Use Without Demonstrations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent work augmented LLMs with tools to mitigate factual inaccuracies and computational errors. |
| Approach: | They propose a method to synthesise tool-use traces using the LLM itself. |
| Outcome: | The proposed method improves performance on a long-tail knowledge task, but not on other datasets. |
Are We Done with MMLU? (2025.naacl-long)
Copied to clipboard
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, Pasquale Minervini
| Challenge: | MMLU is widely adopted but its ground truth errors obscure the true capabilities of LLMs. |
| Approach: | They propose a framework for identifying dataset errors using a novel error annotation protocol and a subset of 5,700 manually re-annotated questions. |
| Outcome: | The proposed framework is based on 5,700 re-annotated questions from the MMLU benchmark. |
GRADA: Graph-based Reranking against Adversarial Documents Attack (2025.emnlp-main)
Copied to clipboard
Jingjie Zheng, Aryo Pradipta Gema, Giwon Hong, Xuanli He, Pasquale Minervini, Youcheng Sun, Qiongkai Xu
| Challenge: | Retrieval Augmented Generation (RAG) frameworks are susceptible to adversarial attacks that manipulate the retrieval process by introducing documents that are adversarially similar to the query. |
| Approach: | They propose a framework that integrates external retrieval modules into RAG frameworks to improve the factual accuracy of large language models. |
| Outcome: | The proposed framework reduces adversarial attacks by 80% while maintaining minimal loss in accuracy. |
DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations (2025.findings-emnlp)
Copied to clipboard
Aryo Pradipta Gema, Chen Jin, Ahmed Abdulaal, Tom Diethe, Philip Alexander Teare, Beatrice Alex, Pasquale Minervini, Amrutha Saseendran
| Challenge: | Large Language Models often produce unfaithful or factually incorrect outputs . masking retrieval heads can induce hallucinations, but decoding by contrast can reduce hallucinosity . |
| Approach: | They propose a training-free decoding strategy that contrasts the outputs of the base LLM and the masked LLM. |
| Outcome: | The proposed decoding strategy reduces hallucinations by contrasting the outputs of the base and masked LLMs. |